We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). More than 800 participants from 13 cities worldwide performed these activities in 131 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,422 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions -- including a novel "expert commentary" done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources will be open sourced to fuel new research in the community.

We address two domains of skill transfer problems encountered by an autonomous robot: within-domain adaptation and cross-domain transfer. Our aim is to provide skill representations which enable transfer in each problem classification. As such, we explore two approaches to skill representation which address each problem classification separately. The first representation, based on mimicking, encodes the full demonstration and is well suited for within-domain adaptation. The second representation is based on imitation and serves to encode a set of key points along the trajectory, which represent the goal points most relevant to the successful completion of the skill. This representation enables both within-domain and cross-domain transfer. A planner is then applied to these constraints, generating a domain-specific trajectory which addresses the transfer task.